Adaptive critic designs: A case study for neurocontrol

-For the first time, different adaptive critic designs (ACDs), a conventional proportional integral derivative ( PID) regulator and backpropagation of utility are compared for the same control problem---automatic aircraft landing. The original problem proved to contain little challenge since various conventional and neural network techniques had solved it very well. After the problem had been made much more difficult by a change of parameters, increasingly better performance was observed by going from the simplest ACD to more sophisticated designs, dual heuristic programming having been ranked best of all. This case study is of use in general intelligent control problems for it provides an example of the capabilities of different adaptive critic designs. Keywords---Adaptive critic, Heuristic dynamic programming, Aircraft autolanding, Neurocontrol, Neural networks for control and optimization. 1. A D A P T I V E C R ITIC D E S I G N S ACDs are tools to solve difficult optimization problems (Werbos, 1990). They include heuristic dynamic p rogramming (HDP), dual heuristic programming (DHP) and globalized D H P (GDHP) as well as their action dependent forms, further referred to as having prefix AD. (We will use the four abbreviations: H D P , DHP, G D H P , and ACD, plus the prefix A D on the first two, for the remainder of this paper.) All designs at tempt to approximate dynamic programming. This gives ACDs their power to work effectively in a noisy, nonlinear environment while making minimal assumptions regarding the nature of that environment. A typical A C D consists of three neural ne t s critic, action and model. The critic net outputs a function J which is an approximation of the Acknowledgements: The authors I gratefully acknowledge support from the Texas Tech Center for Applied Automation and Research Grant entitled "Applied Computational Intelligence Laboratory", and the National Science Foundation Neuroengineering Program Grant No. ECS-9413120. DHP and related designs are patents pending 1994 by BehavHeuristics, Inc. and Paul Werbos. All rights are reserved. The authors also would like to thank Paul Werbos and Ken Otwell for their support and assistance. Furthermore, we are pleased to acknowledge the work of Chuck Jorgensen et al. in the original development of the autolander problem statement (Jorgensen and Schley, 1990). Requests for reprints should be sent to Danil Prokhorov, Box 43102, Department of Electrical Engineering, Texas Tech University, Lubbock, TX 79409-3102, USA. secondary utility function J* of dynamic programming (as in HDP, A D H D P , and G D H P ) or the derivatives of J* with respect to the state variables R (as in D H P and ADDHP) . This function has to be approximated because of intractable computat ional complexity in finding it. The goal is to maximise or minimize J in the immediate future (in the next time step) which produces an op t imum U, the pr imary utility function, in the long run. This goal is accomplished by the action network which outputs a control vector A that optimizes J. An adaptat ion of the action network is based on derivatives of J with respect to the components of the vector A. A straightforward way to get those derivatives is to use backpropagat ion through the other nets of the design. The use of the backpropagat ion algorithm to find the derivatives of J is the most crucial distinction between H D P and the well-known adaptive critic element (Barto et al., 1983). I f the environment or an object to be controlled does not allow backpropagat ion through itself, the model network is used. It mediates propagat ion of the derivatives of J from the critic network to the action network in H D P and, additionally, provides proper targets for the critic network 's adaptat ion in DHP. The model network forecasts the next states R ( t + l ) , R ( t + 2 ) , . . . , R ( t + N ) of the environment. These forecasts, rather than actual states, are fed into the critic provided that the model is sufficiently accurate. Whenever making one-step-ahead predic-